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Abstract 


In  this  paper  we  consider  the  optimal  control  structure  for  multi-shop 
(Part  I:  Parallel,  Part  II:  Series)  systems,  where  the  input  to  the  shop 
system  is  random  and  the  shop  output  is  determined  by  the  number  of  workers 
in  the  shop.  The  number  of  workers  available  to  the  system  is  held  constant, 
while  control  is  exercised  in  discrete  time  by  adjusting  the  allocation  of 
workers  to  the  various  shops  In  the  system.  There  Is  a cost  for  transferring 
workers.  Additionally,  there  is  a cost  of  holding  backlog  in  the  system. 

The  control  objective  Is  to  minimize  the  sum  of  these  costs  over  an  infinite 
horizon. 

It  Is  shown  that  for  some  "regions"  of  the  system's  state  space  the 
optimal  control  policies  are  known  exactly  without  resorting  to  computational 
methods.  For  other  "regions"  It  is  shown  that  the  problem  can  be  decomposed 
Into  subproblems  of  reduced  complexity.  Finally,  an  inertia  (hysteresis) 
property  Is  established  ^Ich  reduces  the  number  of  policy  combinations  which 
must  be  considered  in  some  cases,  and  completely  eliminates  the  necessity 
to  determine  policy  In  other  cases.  The  net  result  Is  a substantial  reduction 
In  the  computer  storage  and  computational  effort  required  to  solve  for  the 
optimal  control  policy. 
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PART  I 
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Introduction 

Consider  a multi-shop  system.  Work  (measured  In  man-hours)  arrives  at 
the  system  periodically,  say  once  a week.  The  amount  of  work  arriving,  to 
each  shop  per  period  Is  assumed  to  be  random  variable  which  can  be  described 
by  a stationary  probability  density  function.  The  amount  of  work  processed 
out  of  a shop  during  a period  Is  proportional  to  the  number  of  workers  In 
the  shop.  The  number  of  workers  In  the  system  Is  constant.  Control  can 
be  exercised  by  adjusting  the  allocation  of  workers  to  the  various  shops 
i In  the  system. 

A multitude  of  paper's  have  dealt  with  the  characteristics  of  shops 
(queues)  under  output  control.  Howard  [14],  Wolfe  and  Dantzlg  [26],  and 

Manne  [18]  developed  numerical  techniques  to  establish  optimal  control  \ 

I 

policies  for  Markov  processes.  The  method,  although  general,  established  ^ 

the  basic  analytic  structure  for  determining  optimal  control  policies  for 
Markov  queues  with  variable  servers.  Many  other  researchers.  Including 
Heyman  [13  ],  Sobel  [24],  Yadln  and  Naor  [27],  Magazine  [16],  [17],  McGill 

[19],  Crablll  [7],  [8],  Beja  [3],  Blackburn  [6],  Balachandran  [1],  [2], 

Rata  [22],  Reed  [23],  Faddy  [10],  [11],  Deb  [9],  Bell  [4],  [5],  Winston  [25], 

Odkenyl  [20],  [21],  and  Llppman  [15]  have  contributed  to  the  field.  The 
shop  system  considered  here  Is  unique  In  that  man  power  (l.e.,  a server)  Is 
treated  as  a shared  resource  available  to  several  shops  (queues) . 

The  motivation  for  this  study  comes  from  observations  of  a Naval  Air 
Rework  fhclllty  (NARF) . A NARF  Is  a large  Job  shop  devoted  to  depot  level 
repair  of  naval  aircraft  and  aircraft  spare  parts.  NARK  maintain  both 
extensive  work  standards  on  all  jobs  performed  and  extensive  data  systems 
which  pinpoint  each  job's  location  In  the  shop  system.  As  a consequence. 


1 


at  any  time,  the  amount  of  work  backlog  In  any  shop  (In  man-hours) 
is  known.  Control  of  the  shop  backlogs  can  be  affected  by  transferring  workers 
between  shops.  This  is  generally  easier  than  authorizing  overtime  from 
managements'  point  of  view. 

There  is  a cost  of  maintaining  backlog  in  the  system  which  is  roughly 
proportional  to  the  amount  of  backlog.  The  backlog  in  the  NARF  shops  can't 
be  used  by  the  fleet  as  spare  parts.  As  a consequence,  the  higher  the  NARF 
shop  backlogs,  the  higher  the  Navy's  investment  in  spares  must  be.  There 
is  also  a cost  (primarily  administrative)  of  transferring  a worker  from  one 
shop  to  another.  Clearly,  the  act  of  transferring  workers  implies  that  the 
worker  Involved  has  the  prerequisite  skills  ^or  that  shop  to  which  he  is 
transferred.  It  follows  that  a NARF,  for  control  purposes,  can  be  broken 
into  sub-systems  of  shops  with  similar  worker  skills. 

There  is  no  claim  here  that  this  description  of  the  shop  control  problem 
of  a NARF  takes  into  account  all  facets  of  what  is  an  extremely  complex 
real-life  control  problem.  It  does,  however,  capture  the  essence  of  the 
system.  In  this  paper,  an  analytic  model  of  a shop  system  is  developed. 

The  system  is  modeled  as  a discrete  time  Markov-Decislon  process.  Character- 
istics of  the  optimal  control  structure  for  the  system  are  derived. 

In  Part  1,  we  approach  the  parallel  shop  case.  In  Part  2,  the  series 
shop  case  is  considered.  In  both  cases,  the  development  is  for  two  shop 
systems.  The  extensions  to  N-shops  are  straight-forward,  but  lengthy.  The 
Interested  reader  is  referred  to  Haas  [12]  which  is  available  from  the 
authors. 


The  Two  Shop  System  (Parallel 


Let  us  now  address  the  problem  of  determining  general  properties 
for  the  optimal  control  structure  of  a system  with  2 parallel  shops  which 
are  controlled  by  transferring  workers  from  one  shop  to  the  other. 

At  equally  spaced  points  In  time  (surge  points)  a quantity  of  work 
may  enter  the  shop.  These  quantities,  measured  In  work  units,  are  Independ- 
ently, but  not  necessarily  Identically,  distributed  random  variables.  Work 
which  cannot  Immediately  be  processed  Is  held  as  backlog  for  the  shop  It 
entered.  The  transfer  of  backlog  from  one  shop  to  another  Is  prohibited. 

In  addition,  the  flow  of  work  through  each  shop  does  not  pass  through  the 
other  shop.  Thus,  the  system  may  be  thought  of  as  two  shops  In  parallel 
(See  Figure  1) . 

The  total  number  of  workers  In  the  system  Is  W.  A worker  may  be 
transferred  at  specific  points  In  time  (those  points  to  be  defined  shortly) 
from  one  shop  to  the  other  at  a cost  of  T.  Each  of  the  W workers  (while 
working  In  shop  1)  produce  work  units  over  an  Interval  of  time  between 

two  consecutive  Input  surges.  Note  that  the  couching  of  Input  in  terms  of 
'work  units'  rather  than  'man-hours'  and  using  an  efficiency  rating  of 
allows  for  differences  In  shop  operational  characteristics. 

There  Is  a backlog  holding  cost  of  H per  unit  Incurred  Immediately  before 
a surge  point  to  the  shops.  So,  If  Is  the  backlog  In  shop  1 Immediately 
before  Input  surge  n,  the  total  backlog  holding  cost  In  period  n Is  H(B^  -t*  B2) 

Control  of  the  system  Is  exercised  by  transferring  workers  from  shop 
to  shop  Immediately  after  each  surge  point  (see  Figure  2 where  the  transfer 
of  a worker  from  one  shop  to  the  other  results  In  a change  In  the  slope  of 
the  backlog  function).  The  objective  Is  to  exercise  control  so  as  to  minimize 
the  total  of  holding  and  transfer  costs  over  some  horizon. 
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SHOP  I 
BACKLOG 


Figure  2,.  Realization  of  backlog  function 


If  the  process  is  imbedded  immediately  after  each  surge  point,  it 
can  be  formulated  as  a Markov-Dee is ion  process.  In  order  to  formulate  the 
optimization  as  a dynamic  program,  the  state  of  the  system  at  stage  R is 
defined  as  a four  dimensional  vector  (B,  d) . This  vector  may  also  be 
written  as  (Bj^,  B2,  dj^,  d2) . where  B^  is  the  backlog  in  shop  i at  stage 
K and  d^  is  the  number  of  men  in  shop  i at  stage  K + 1.  It  is  easy  to 
convince  one's  self  that  the  state  definition  satisfies  the  Markov  property. 
Before  continuing,  the  following  two  definitions  are  required: 

(^(B,  d)  =■  the  minimum  expected  discounted  cost  over  an  infinite 

horizon,  given  that  there  are  now  B units  in  the  backlogs 
and  there  were  d men  in  the  shops  during  the  previous 
transition  (0  ^ B^  < »,  d^  = 0,  1,  2,  ....  W,  i =■  1,  2). 

f^(m|B)  dm  > the  probability  that  the  backlog  at  the  end  of  the 

transition  is  between  m and  m + dm  given  that  backlog 
at  the  beginning  of  the  transition  is  B and,  that  policy 
d is  in  effect  during  the  transition  (0  ^ m^  < <») . 

Let  T be  the  cost  of  transferring  a worker  from  one  shop  to  another. 

Let  p ■ (Pj^,  P2)  be  the  present  worker  allocation  and  d ■ (d^^,  d2)  be  the 
worker  allocation  during  the  previous  transition.  The  recursive  equation 
can  now  be  written.  It  Includes  the  transfer  cost  and  backlog  holding  cost 
for  the  Immediate  transition  plus  the  discounted  expected  cost  over  an 
infinite  horizon.  Thus, 
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The  Structure  of  the  Optimal  Control  Policy 


In  this  section  four  fundamental  properties  of  the  optimal  control 
policies  for  an  Infinite  horizon  will  be  determined.  These  properties 
will  allow  one  to  determine  a great  deal  about  the  structure  of  the  optimal 
control  policy. 

Property  1: 

The  first  property,  stated  and  proved  In  the  theorem  below,  shows 
that  when  shop  backlogs  are  high  enough  the  optimal  policy  Is  known 
Immediately.  Before  beginning  the  proof  for  property  1,  two  necessary 
lemmas  will  be  established. 

Lemma  1 

+ 0,  B^,  d)  - $(B^,  B^,  d)  ^ . 

(The  same  property  holds  In  B2.) 

Proof: 

Take  two  starting  points  In  backlog  space. 

Point  1 ■ (B^,  B2.  d)  and  Point  2 =•  (B^  +9,  B^,  d) . 

For  both  points  take  a specific  Input  realization.  For  that  Input 
realization  and  starting  Point  1,  there  are  a set  of  optimal  policies 
which  will  be  enforced  at  each  stage.  For  the  same  Input  realization, 
assume  that  these  policies  will  be  enforced  for  the  sequence  of  points 
starting  at  Point  2.  Then  for  each  stage  the  separation  between  the 
sequence  of  points  starting  at  Point  1 and  those  starting  at  Point  2 Is 
no  more  than  9.  Thus,  differences  In  holding  costs  at  each  stage  are 
no  more  than  H6.  Transfer  costs  are  equal  at  each  stage.  For  the  Infltlte 
horizon  this  Implies  that  the  total  difference  In  cost  must  be  less  than 
or  equal  to. 


Imt. 


* . H • e 

H • I aie  - 

i-0 

The  actiial  expected  cost  under  optimal  control  must  be  less  than  or 
equal  to  this.  Th\is  the  lemma  must  hold  (It  should  be  noted  that  In 
the  limit  (as  -*■  <»)  It  Is  easy  to  see  that  the  lemma  Inequality  becomes 


an  equality) . 


Q aE  • D • 


Lemma  2: 


Proof : 


1<KB,  d)  - <KB,  d')|  < |d^  - d||T 


"yCB,  p)  * aJI  f^(m|B)  $(m,  p)dm 


Let  p be  the  optimal  policy  for  (B,  d) . If  p Is  made  the  policy  for 
(B,  d'),  then  the  expected  discounted  return  for  that  state  Is  just, 

4^(8.  d’)  - TIpj^  - d'l  + H(B3^  + B^)  + ’P(B,  p) 

< T|p^  - d^l  + T|d^  - d'l  + H(B^  + B2)  + 'F(B.  i) 


< Tldj^  - d^l  + 4(B,  d). 

The  superscript,  p,  on  4^(B,  d')  Indicates  that  p,  the  policy  for  which 
the  retiim  Is  calculated,  may  not  be  optimal  at  (B,  d’).  Rearrangement  of 
the  above  gives. 


T|d^  - d^l  > 4^(3,  d’)  - 4(B,  d) 


since 


4^(5,  d')  > 4(B,  d’). 


this  implies  that. 


T|<i^  - d^l  1 *(8,  d')  - 4(B.  d) 


A similar  argument  shows  that. 


Hence, 


Tl<i{  - dj^l  1 «(B.  5)  - <KB,  d’) 


T|<ii  - djl  1 I <tCB,  d)  - -KB,  d')l. 


Q.E.D. 


With  the  foregoing  results  established,  one  may  prove  the  existence 
of  property  1 for  the  Infinite  horizon. 

Theorem  1:  Let  R2  > Define 


(la) 


(lb) 


n * n^  + n2,  and 


0 - [T  + n^R2H/(l  - a)]a 
1.  When 

T < H(R2  - R3^)a/(1  - a)^. 


If  n2  Is  selected  such  that 


(Ic) 


^2-  h 


2R2  - Ri  - 


> a 


(Id) 


(le) 


(If) 


n,+l  n„ 

H(R2  - R^)a(l  - a ^ ) a ^ R2H 

^ “ T > 


(1  - a) 


(1  - a) 


j,  smd 


nz-l 


T < H(R  - IL)  I jo 
j-1 


j 


Then  when 


n2-l 


^ H(R«  -i  R.)  'I  ja'^  - 9,  and 
j-1 
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^ n • R.  * W,  It  is  optdjoal  to  place  all  workers  in 


2.  When 


Then  when 


In  other  words  when 


There  will  always  be  an  n such  that  there  is  a region  B . ^ n • R • W 


for  all  i where  it  is  optimal  to  maintain  the  worker  allocation  from  the 


previous  stage. 

Furthermore, 

since  as  n.  -•-  <»  the  term  in 

(If),  1 ^ 2 

• 

j-1  (1  - a) 

when 

i 

i 

T < 

H(R2  - Rj^)a(l  - o)^ 

/■ 

’ 

10 

- - - 

i 


there  will  always  be  an  n large  enough  that  there  is  a region 

^ n * * W where  It  Is  optimal  to  place  all  workers  In  shop  2. 

Proof; 

The  method  for  completing  the  proof  will  be  as  follows: 

It  will  first  be  assumed  that  It  Is  never  optimal  In  the  first 
n^  stages  to  decrease  the  number  of  workers  In  shop  2,  and  that  for  the 
first  n^  stages  It  Is. only  optimal  to  Increase  the  number  of  workers  In 
shop  2 at  the  first  stage. 

In  part  1,  It  will  be  shown  that  given  the  above  assumption  and  given 
that  (11)  holds  It  Is  always  optimal  to  maintain  the  nimiber  of  workers  in 
the  previous  stage.  In  part  2 (also  given  the  above  assumption),  the 
optimality  of  a complete  transfer  of  workers  to  shop  2 will  be  shown  for 
the  case  where  (If)  holds.  In  part  3,  the  proof  will  be  completed  by  dem- 
onstrating that  the  initial  assumption  must  hold. 

Part  1;  Assume  that  Inequality  (11)  holds.  Fix  a specific  Input  realization. 
The  expected  cost  when  no  policy  change  is  made  for  n^  transitions  is 


(2) 


*’(B,  p)  - cj;  + a 


idiere  C Is  expected  discounted  holding  cost  up  through  n.  transitions  and 
“l  ^ 

Cjj  ^2^  is  the  expected  discounted  cost  (under  optimal  control)  after  n^^ 

transitions.  By  assumption,  there  Is  no  transfer  cost,  , through  the 

“l 

first  n^  transitions. 

The  expected  cost  (when  a transfer  of  X workers  from  shop  1 to  shop  2 
Is  made  at  the  current  stage,  the  allocation  Is  held  constant  over  n^  stages, 
and  the  system  Is  optimally  controlled  thereafter)  Is, 
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'v 


$"(B,  p)  - C^'  + c|;'  + a\:  . 


where  Is  the  expected  discounted  transfer  cost  through  n^  transitions 
C^' 

n^  is  the  expected  discounted  holding  cost  throxigh  n^  transitions,  and 

t 

C .1  is  the  expected  discounted  cost  (under  optimal  control)  after  n. 

Uj^+l  1 

transitions . 

For  any  Input  realization  there  can  be  no  more  than  n^  X separating 
the  Individual  shop  backlogs  for  one  control  policy  (p)  versus  the  other 
(p')  after  n^  stages  of  the  process.  Thus, 

l\+l-  * *!•  ®2  - ®2-  - "‘'l-  '2-  5’)l 


< + e^,  - 02,  P)  - B2,  p)| 

+ l$(Bj^.  B2,  P)  - <t(Bj^,  B2,  p')l. 

— ®1  — ”l^®l 


^ 02  1 “1XR2 


Using  Lemma  1,  It  Is  easily  shown  that 


n.R-XH 

|«(B^  + 0^,.B2  - 02.  P)  - p)|  < -f~~’ 


It  Is  also  known  from  Lemna  2 that. 


h(Bj^.  B2.  p)  - <KB^,  B2,  p*)|  < Ipj  - p^It  < 


Thus  from  (4) , (5) , and  (6) 


n, R,XH 
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Rearranging  (la) 


_ - ^1^2^  e 

1 - a n. 


Substituting  (8)  Into  (7), 


C « cl  < 

nj^+1  n^^+l  ” nj^ 

a 


Removing  the  absolute  value  sign  and  multiplying  by  a gives, 

(9)  i X6  + »°1 

h f 

Now  consider  the  relationship  of  C and  C . Since  neither  shop  can  run 

“l  “l 

n -1 

h i li  * 

out  of  work,  Is  XHCR^  “ Rj^)  1 greater  than  C . Thus 


(10)  cJl  - C*;'  + XH(R,  - R.)  I ja^ 

“l  “l  2 'I 

Adding  (9)  and  dO)  together  yields. 


C**  + a ^ C 


V+l  1 X(0  + H(R-  - R.)  I ja^)  t 
1 ^ 


“l 

+ a ^ C' 


r 1 2 

Using  Inequality  (11)  and  noting  that  ^ ja'*  ■ a/(l  - o)  , 

j-1 


h Vi  * 

C + o ^ C . , < XT  + C + a C 

nj^  nj^+1  — nj^ 


nj^+l‘ 
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r 


t' 

But  since  C > XT  It  is  Immediate  from  (11)  that, 

“l 

h u I 

C + a C . T < C*"  + C + o C ' , , 

^ ~ ®1  ’'^1  Hj^+l 

or  simply 

$'  (B,  d)  < $• ' (B,  d)  . 

Part  2;  Assume  that  inequality  (If)  holds. 

For  this  part  of  the  theorem  the  definitions  for  (2)  and  (3) 
still  hold.  The  argument  given  in  part  1 showed  that 

xe 


Thus, 


(12) 


^ ‘•a 


“l  “l 

“ C . , < 9X  + o C . 

Uj^+l  — nj^+1 


Since  shop  2 cannot  run  out  of  work  through  transitions. 


I °1~^ 

(13)  C^'  1 cl;  - XH(R,  - R^)  I jo^ 

/ X j_'i 


(Note:  if  shop  1 does  not  riin  out  of  work  during  the  first  n^  stages, 
the  above  Inequality  becomes  an  equality.) 

Adding  (12)  and  (13)  gives 


n,-l 

n.  ^ h 

C“  + o C'  , 1 X(9  - H(R  - R,)  I ja^)  + C + a C . . 


,h’ 

n. 


but  (If)  requires  that: 


Ui-l 


T < H(R,  - R^)  [ - 9 

^ ^ j-1 
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Thus 


XT  + C^'  + o 


or  simply. 


Uj^+l  — 


< C 


+ a ^ C 


Uj^+l 


f"(B,  d)  £ 4"(B,  d). 

Part  3;  The  proof  Is  completed  by  showing  that  the  optimal  policy  at  stage 
1 will  remain  optimal  through  the  first  n^  stages  and  by  showing  that  It  Is 
never  optimal  to  decrease  the  number  of  workers  In  shop  2.  This  will  be 
shown  by  considering  two  cases  separately. 

Case  1:  In  case  1 It  will  be  shown  that  when  the  conditions  In  the  proof 
statement  hold  It  Is  never  optimal  In  the  first  n^  stages  to  decrease  the 
number  of  workers  In  shop  2. 

Let  the  optimal  policy  at  stage  Z be  to  transfer  X workers  from  shop  2 
to  shop  1.  If  d “ (dj^,  d2)  Is  the  policy  of  the  previous  stage,  the  policy 
at  stage  Z Is  (dj^  + X,  d2  - X) . The  system  is  optimally  controlled  thereafter. 
The  shop  system  with  this  control  policy  will  hereafter  be  designated  option  1. 

Consider  a second  system  which  Is  Identical  to  the  first  except  that  the 
policy  In  force  at  stage  Z Is  (d^,  d2) • All  policies  before  and  after  stage  Z 
are  Identical  to  those  of  option  1.  Call  this  second  system,  option  2. 

Now  fix  an  Input  realization  and  compare  the  expected  return  for  the  two 
options.  It  will  be  shown  that  option  2 provides  the  lowest  expected  return, 
and  therefore  that  the  postulated  decrease  Is  never  optimal. 

For  option  2,  since  shop  2 will  not  run  out  of  work  for  at  least  the  first 
n^  + 02  stages,  the  holding  cost  Is  at  least 

nj^+n2-Z 

XH(R  - IL)  I j+Z-1 

2 ^ j-1 
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less  than  for  option  1.  Transfer  costs  for  option  2 are  less  than  or  equal 
to  those  for  option  1.  Thus, 


ni-h.j-2 


(14) 

where 
-h' 


“l'*^2  “l'^2  ~ “l'^2  “l^2  ^ ^ j“l 


“l'*^2  " 2 holding  cost  for  the  first  n^+  n^  stages 


f.  t 

C . ■ Option  2 transfer  cost  for  the  first  stages 

“l'^2 

, ■ Option  1 holding  cost  for  the  first  n.+n„  stages 

“1-^2  ^ ^ 

. * Option  1 transfer  cost  for  the  first  a,+n_  stages 

At  the  end  of  if  the  system  state  for  option  1 is  (B^,  d)  , 

the  system  state  for  option  2 will  be  (B^^  - ^2  + XR^,  d)  , 0 ^ X.  Lemma  (1)  can 

be  used  to  show  that  the  mavinnim  possible  difference  in  expected  cost  for  the 
ensuing  stages  will  be 

Oi+nj 


Thus  if 
C' 


+n»+l 


“1-^2 


Option  2 total  expected  cost  for  stages  nj^+n2+l  on 


C ■ Option  1 total  expected  cost  for  stages  n-+n-+l  on 

nj^+n^+l  ^ ^ 


(15) 

Combining  (14)  and  (15)  gives 


n +n 

a ^ ^ XR-H 

d*  • 


pt  < r 4*  I ■ I 

^n^+nj+l  - nj^+ti2+l  1 - a 
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(16) 


C***^-.  + + c'  < c*'  . + . + c ... 

“l'^2  “l'^2  “j^+n2+l  “ “l'^2 


n +n»-Z 

- XH(R  - IL)  I + # r-  XR„H. 

j-1 


1 - a 


Now  from  (16),  so  long  as 


“l+"2 


n^+n^-Z 


0 > 


- l-a  ®2H  - JHCEj  - Rj)  . 


total  option  2 expected  cost  will  be  less  than  that  for  option  1.  Certainly 
this  will  be  true  for  any  policy  changes  occurlng  In  the  first  n^  stages  If 


“1-^2 


0 > 


-1-0 
Or  simplifying  If, 


XR«H 


- XH(R  - R ) I a 
^ j»l 


j-Hii-l^ 


n_  n.. 

„ 2 2 

0 > R„  - (R-  - R^)  I ctJ 

^ j-1 


-1-0  *"2 


But  this  Is  equivalent  to  condition  (Ic)  In  the  theorem  statement. 

Case  2;  Now  to  conlude  the  proof  It  will  be  shown  that  It  Is  never  optimal  to 
Increase  the  number  of  workers  In  shop  2 at  any  but  the  first  stage.  Asstime 
that  at  some  stage  Z (not  the  first  stage)  the  policy  Is  to  Increase  the  number 


of  workers  In  shop  2 by  X.  So  that  If  d * (dj^,  d2)  Is  the  policy  of  the  pre- 


vious stage  the  policy  at  the  current  stage  Is  (dj^-X,  d2+X) . The  system  Is 
optimally  controlled  thereafter.  The  shop  with  this  control  policy  will  be 
designated  option  1. 

Consider  a second  system  with  a policy  at  the  previous  stage,  Z-1,  of 
(dj^-X,  d2+X)  and  policies  which  are  the  same  as  option  1 for  all  stages  before 
and  after  Z-1.  Now  fix  an  Input  realization  and  compare  the  expected  return 
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for  the  two  options.  Let  the  system  costs  for  this  case  take  the  same  symbols 
as  those  for  costs  in  Case  1. 


It  is  Immediately  clear  that 


(17) 
Furthermore, 


C^'  - (1  - o)T. 

1^2  “l^2 


nj^+n2“Z+l 

^-'+n  - ^ -hi  ■ ^ 

ni^2  “1^2  2 1 j.i 

Also, 


(19)  C 


“l^2 


< - 


XR,H 


lU+ii.+l  - n,+n,+l  1 - a “ 2 


Adding  (17),  (18),  and  (19)  gives 

(20)  C^'  + C^'  + C'  . ^ +■  C 

nj^+n2  “l'^2  ~ “l^2  ”l^2 


+ Xa^“^  (1  - o)T  + “ 


ni+n2 


1-0 


XR 


2H  - XR(R2  - R^)  I 


nj^+n2-Z+l 


j-1 


Now  from  (20)  , so  long  as 


“I'^Z 

(21)  0 > o^'^d  - o)T  + ^ 


1-0  “2 


n.+n2“Z+l 

R„H  - H(R2  - R^)  I 


j-1 


total  option  2 expected  cost  will  be  less  than  that  for  option  1. 
Inequality  (21)  may  be  rearranged  to  read 


1 0^“^  [(1  - oi)T  - H(R2  - o^]  + R2H 


nj^+n2~Z+l 


”1-^2 


Note  that 


j+Z-2 
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Oi+nz-Z+l  “l'^2 

[(1  - a)T  - H(R  - IL ) I a^]  + ^ R H 

2 1 j.i  1 - a 2 


n2+l 


ni+U2 


< [(1  - a)T  - H(R  - R ) I ah  + 

j=l  -L  - 


— R„H. 
a 2 


Note  from  the  conditions  of  the  theorem  statement  (i.e.,  (lb)  and  (Ic) , when 


OB 

(1  - a)T  - H(R  - IL.)  J < 0 

j=l 


n^  must  be  selected  so  that 


n2+l 


(1  - o)T  - H(R  - R^)  I < 0. 
^ ^ j“l 


Thus, 


[(1  - a)T  - HCR^  - R^)  I a^]  + ”y'_  ^ V 


n^+l 


“1-^2 


.2+1 


.1+112 


< a ^ [(1  - a)T  - H(R  - IL)  I .^1+^ 

X j_l  X - a 


R2H. 


To  simnnarlze,  this  argument  shows  that  if  n^^  and  n2  are  selected  so  that 


(lb)  and  (Ic)  hold,  then 
_Z-2 


.1+.2 


(1  - a)T  + 


1-0 


RH 


- H(R2  - R^)  I 


n^^+n^-Z+l 


j+Z-2 


j-1 


.2+1 


"1^2 


< a ^ [(1  - a)T  - H(R2  - R^^)  a^]  + ^ R^H  < 0 


This  means  that  option  2 expected  cost  is  less  than  option  1 expected  cost. 
Thtis  when  (lb)  and  (Ic)  hold  it  is  never  optimal  to  Increase  the  number  of 
workers  in  shop  2 at  any  stage  but  1 (or  if  it  is  optimal  to  increase  the 
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number  of  workers  at  a stage  other  than  1«  an  alternative  optimal  policy  exists 
at  that  stage  which  does  not  Increase  the  number  of  workers  In  shop  2.) 

When  (le)  does  not  hold  then  of  course, 

T > H(R2  - Rj^)a/(1  - a)^ 

In  this  case  the  nonop tlmallty  of  a shop  2 worker  Increase  at  any  stage 
other  than  1 can  be  shown  with  a similar  argument.  Here  such  an  Increase  Is 
postulated  at  stage  Z.  It  can  then  be  shown  that  when  (Ig)  holds  the  expected 
cost  Is  reduced  by  delaying  the  worker  Increase  by  one  stage. 

Q . E .D. 

Property  2; 

Just  as  backlog  can  be  so  large  that  an  optimal  policy  at  stage  k Is  known 
Immediately,  backlog  levels  may  be  so  low  that  the  optimal  policy  Is  known 
Immediately . 

Before  proving  this  property,  It  Is  necessary  to  note  that  the 
discounted  system  cost  function  Is  Increasing  In 
Lemma  3; 

For  the  two  shop  system; 

$(B^  + 0,  B^,  d)  - <t(Bj^,  B^,  d)  ^H0  and 

B^  + 0,  d)  - 'f(B^,  B^,  d)  ^ H0 
Proof;  The  proof  Is  obvious: 
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Theorem  2;  If  the  state  of  the  system  (B,  d) 


■ (Bj^,  B2,  dj^,  d2)  is  such  that  B^^  ^ d^^  • and 

B2  £ d2  • R2»  the  optimal  policy  is  to  maintain  the 
manpower  allocation  of  the  previous  stage,  d. 

The  above  restriction  on  the  state  space  merely  requires  that  if  the 
manpower  allocation  of  the  previous  stage  is  maintained,  before  the  next 
input  surge  both  shops  will  become  idle.  Given  the  stated  restrictions  on 
the  state  space,  a change  in  worker  allocation  can  only  Increase  the  backlog 
in  the  shops.  Thus,  a change  in  policy  is  premature.  It  is  better  to  wait 
until  one  knows  where  the  system  will  be  after  the  next  input  surge,  and  then, 
if  necessary,  make  a policy  change. 

Proof;  Suppose  that  the  state  of  the  system  satisfies  the  requirements  immediately 
above.  Then,  since  £ d^^  • R^  and  ^2  ^ d2  • R2* 

From  this  and  Lemma  2 it  is  clear  that; 


d)  f^(mj^,  “2!®!’  *^2 


II  ’<"1-  "2- 

> II  *(01^,  m2,  d)  f^(m^,  m2|  B^^,  B2)  dm^^  <^2- 


Also,  from  Lemma  3: 


(23) 


T|Pi  - P)  1 m2,  d). 

Substituting  ( 2^  Into  22  and  multiplying  through  by  a gives 

(24)  oTIpj^  - + a||  m^,  p)  f^(mj^,  m2!B^,  B2)  dm^  dm2 

-“(/  *<v  v d)  f^(mj^,  m2|Bj^,  B2)  dm^  <^2 
Inequality  (24)  Is  unchanged  If  a ^ 1 is  removed  from  the  transfer  term 
In  the  eqtiation  and  H(B^  + B2)  Is  added  to  both  sides.  But  the  resulting 
expression  Is  just  an  Inequality  relating  the  expected  cost  of  the  system  under 
policy  p and  policy  d.  Thus,  d must  be  optimal. 

Q.E.D 

Property  3 (The  Inertia  Property) 

The  third  property  to  be  established  is  the  Inertia  property  for  the  two 
shop  system.  It  Is  in  a sense  a generalization  of  Heyman's  [12]  main  result 
to  the  two  shop  case.  The  Inertia  property  for  the  two  shop  case  is  established 
by  the  following  theorems: 

Theorem  3:  Consider  two  distinct  points  In  the  shop  state  space 

(B,  d)  and  (B,  d').  Let  p be  optimal  at  (B,  d)  and  let  p', 

be  optimal  at  (B,  d').  Furthermore,  let  d^  ^ d^  ^ p^^  and 

dj^  ^ pj^  or  let  dj^  ^ d|  ^ p^^  and  d^  — P].*  P also  optimal 


at  (B,  d'),  and  p'  Is  optimal  at  (B,  d) . 


Proof : 


Once  again,  let 

'?(B,  p)  ■ ajj  f^(m|B)  $(m|p)dm. 
Since  p Is  optimal  at  (B,  d) , 
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jA 


•1 

j 


f 


f 


I 


(25) 

H(Bj^  + B2)  + 

T|d^ 

- p^l  + ^(B, 

P) 

< H(Bj^  + 

B2>+T|di  - p'l 

+ 'i'(B,  i'). 

Likewise 

i,  since  p'  is  optimal  at  (B,  d') 

(26) 

H(B^  + B2)  + 

T|d' 

- p{|  + ’i'(B, 

p': 

) < H(Bj^  + 

B2)+T|d^  - pj 

+ 'i'(B,  i). 

Observe 

that  when. 

(27) 

■ii  i > Pi 

and  c 

lilPi 

UI  - Pil  - 1 

Pll  + (dl  - 

‘'i> 

and  |dj^  - 

p'l  - |d^  - p'l  + (d* 

When 

(28) 

■>1 1 i Pi 

and  d 

111  Pi 

Ui  - Pil  • 1 

‘^l- 

Pll  - (di  - 

‘'i> 

and  |d^  - 

p;|  - Id,  - Pil  - <di 

Since  and  must  satisfy  either  equation  (27)  or  (2a),  inequality  (26)  may 
be  written. 


(29)  H(Bj^  + B2)  +T|d^  - p^t  + y(B,  p')  < H(Bj^  + B2) 

+ T!dj^  - p^l  + ^(B,  p). 

Inequalities  (25)  and  (29)  imply  that, 

(30)  H(Bj^  + B^)  +TI<Lj^  - p||+  y(B,  p')  - H(Bj^  + B^)  + ijd^^  - p^  | 

+ 'i'(B,  p). 

Since  the  left  hand  side  of  (33)  is  the  expected  return  with  a policy  choice 
of  p';  since  the  right  hand  side  is  the  expected  return  with  p worker  allocation, 
and  since  p is  optimal  at  (B,  d) ; p'  must  be  optimal  at  (B,  d)  as  well.  The 
same  argtiment  can  be  used  to  show  that  p is  optimal  at  (B,  d) . 

Q.E.D. 
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Proof; 


The  proof  Is  executed  by  considering  four  possible  cases. 

Case  1:  Pj^  i S and  pj^  ^ S + Q 

Case  2;  Pj^  £ S and  pj^  ^ S + Q 

Case  3:  £ S and  p^  ^ S + Q 

Case  4:  p^  > S and  p^  £ S + Q 

The  following  arguments  show  that  for  each  case  Pj^  i 
Case  1:  If  S + Q ^ Pj^,  then  p^^  < S + Q ^ pj^.  If  S + Q p^,  then  Pj^  “ p^ 
by  Theorem  3. 

Case  2:  If  S < pj^  then  pj^  ^ S < pj^.  If  S ^ pj^,  then  p^^  ■ p^  by  theorem  3. 

Case  3;  Pj^  £ S < S + Q £ p^ 

Case  4:  For  case  4 the  fact  that  p^  ^ pj^  Is  shown  by  contradiction.  Assume 

the  opposite. 

Pi  > Pi 

Since  p Is  optimal  at  (B,  S,  W - S), 

(33)  H(Bj^  + B2)  + T|p^  - S|  + 4'(B,  p)  < H(Bj^  + B2)  + T|pi  - S| 

+ ’i'(B,  p'). 

Since  p*  is  optimal  at  (B,  S + Q,  W - (S  +Q)) 
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w 


(32)  H(B^  + B2)  + TIpj^  - (S  + Q)  I + 'J'(B,  p)  > H(Bj^  + B^)  + T|p^  - (S  + Q)  1 

+ YCB,  p'). 

With  slight  rearrangement  and  the  elimination  of  H(B^  + B^)  on  each  side  of  the 
Inequalities,  (31)  and  (32)  can  be  written  as. 


(33) 


f(B,  5)  < t|p'  - s|  - T|p^  - S|  + y(B,  i') 


(34)  '•'(B,  i)  > T|p^  - (S  + Q)  I - T|p^  - (S  + Q)  I + vd,  p') 

Subtracting  (33)  from  (34)  gives, 

Ipi  - Si  - - S|  > Ipj^  - (S  + Q)1  - Ip  - (S  + Q)1 

But  It  Is  easily  shown  that  for  case  4 

|p^  - s|  - 1p^  - S|  < |p|  - (S  + Q)|  - |p^  - (S  + Q)|. 

Therefore  a contradiction  exists  and  p^  must  be  less  than  or  equal  to  p^. 

Q.E.D. 

Property  4; 

When  the  backlog  in  one  shop,  say  shop  2,  becomes  very  large  while  the 
backlog  In  shop  1 Is  relatively  low,  certain  simplifications  In  the  control 
structure  occur.  The  optimal  policy  structure  for  the  system  Is  sensitive  only 
to  the  backlog  level  of  the  lightly  loaded  shop  (shop  1) . Consequently,  the 
determination  of  optimal  policies  can  be  achieved  by  determining  optimal  policies 
for  an  eqtilvalent  single  shop  system.  This  result  Is  described  In  the  following 
theorem. 

Theorem  4:  As  B^  becomes  large,  the  optimal  policy  structure  asymptotically 
approaches  an  optimal  control  structure  which  Is  defined  by  a 
single  shop  formulation.  (The  precise  form  for  this  single  shop 
problem  Is  given  after  some  motivating  development  In  the  proof.) 
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1 


Proof. 

The  recursive  equation  for  the  system  Is  of  course, 

(35)  «KB,  d)  - ioin{T|d,  - d' | + H(B.  + B.)  + aj  f‘*'(mlB)  *(5,  d')  dm}. 

0<d’£W  ^ ^ 'o 

Now  consider  the  Integral  In  detail.  If  Is  a random  variable  representing 
the  Input  to  the  highly  loaded  shop  (shop  2)  during  a transition  then. 


m2  - B2  + (dj^  - W)  + 
dm2  " *^2 


V 


In  Lemma  1 it  was  noted  that,  for  m^  sufficiently  large, 

(38)  m2  + 0 d')  - m2,  d')  + 

Using  (38),  (37)  may  be  rewritten  as  follows: 


r , id-:  - W + r J H 

J g ^(m^lBj^)  f2(r2)[«(nij^,  82,  d')  + - ^ ] dm^.  dr2. 


(39) 


(d,*  - W + r J H 


Integrating  through  by  12  and  taking  Independent  terms  out,  (39)  becomes. 


, d'  - W)  H E(r-)  H 

Substituting  (40)  into  (35)  yields 


(41) 


$(B,  d)  - min  {T|dj^  - d^  | + H(B^  + B2) 


r ‘^1 

+ o J g (m^lBj^)  $(m^,  B2,  d')  dm, 
"'o 


(d'  - W)  H E(r  ) H , 

+ ^ + -rr-= \ 

1 - a 1 - a 

Constants  on  the  right  hand  side  of  (41)  have  no  effect  on  the  minimization 
process  so  that.  Insofar  as  policy  determination  is  concerned,  an  equivalent 
expression  is 

d,'  H 


(42) 


'--I- 


«(B,  d)  - min  {T|d  - d’  | + H(B.  + BJ  + ^ 

0<d'<  -L^l-o 

+ a j*  g ^(m^|B^)  $(m^,  B2,  d’)  dm^}. 
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In  this  "imaginary  system"  control  is  exercised  by  opening  or  closing  a worker's 
station.  Note  that  what  the  conversion  to  a single  shop  formulation  does  is 
translate  the  holding  cost  for  backlog  in  shop  2 into  a cost  of  employing  a 
server  in  shop  1. 

Q . E.  D . 

Property  5: 

One  property  of  the  optimal  control  system  which  at  first  seems  obvious, 
is  as  follows: 

At  any  stage,  (for  a given  number  of  workers  at  the  previous  stage)  the 
optimal  number  of  workers  in  shop  1 either  remains  the  same  or  Increases  as 
the  backlog  in  shop  1 is  Increased  and  backlog  in  the  other  shop  is  held 
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constant,  this  conjectured  property  will  hereafter  be  called  the  "increasing" 
property.  Conversations  with  other  researchers  have  revealed  that  this  is  a 
topic  of  considerable  Interest. 


Despite  its  intuitive  appeal,  the  increasing  property  for  single  shop  sys- 
tems does  not  always  hold.  In  Haas  [12]  a counter  example  is  given  for  a non 
stationary  single  shop  system  with  transfer  costs.  Winston  [25]  proved  that 
the  increasing  property  does  hold  for  a one  shop  (queue)  stationary  system 
which  Includes  a control  option  of  0,  1,  or  2 workers  (servers)  and  transfer 
costs. 

For  stationary  undiscounted  parallel  shop  systems  our  computational  exper- 
ience has  shown  that  the  increasing  property  rarely,  if  ever,  falls  to  hold. 
However,  for  the  discounted  system,  it  is  possible  to  find  cases  where  the  in- 
creasing property  does  not  always  hold  (see  discussion  in  next  section) . For 
series  shops,  counter  examples  to  the  Increasing  property  are  abundant  even  for 
undiscounted  systems  (see  Part  2:  Series  Shops).  If  there  are  shop  (queueing) 
systems  where  the  optimal  control  rules  can  be  shown  to  exhibit  the  "increasing 
property,"  we  suspect  that  the  systems  are  parallel  (as  opposed  to  series) 
systems  and  that  discounting  is  not  used  in  the  cost  structure  of  the  system. 


Geometric  Interpretation  and  Computational  Efficiencies 

With  these  properties  established  for  the  2-shop  infinite  and  finite  hori- 
zon case,  it  is  possible  to  indicate  geometrically  the  nature  of  the  optimal  con- 
trol structure.  Figure  3a  shows  an  example  structure  for  the  case  where  the  man- 


power allocation  (5  workers  in  the  system)  for  the  previous  stage  was  d > (3,  2) 

,2 


and  lidiere  T > H(R2  - Rj^)  a/(l  - o)  . The  rectangular  region  with  one  corner  at 
the  origin  is  the  "do  nothing"  region  required  by  property  2.  The  other  "do 
nothing"  region  is  a consequence  of  property  1.  At  relatively  large  values  of 
and  B2,  the  position  of  the  Isopols  can  be  determined  by  the  one  shop  formu- 
lation described  in  property  4.  For  heavily  discounted  systems  (on  the  order  of 
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Graphical  representation  of  optimal  control 
structure  for  2-shop,  5-worker  system  when 
the  previous  policy  is  (3,  2)  and 
T < H(R2  - Rj)a/(1  - 0)2. 


Figure  Jb: 


Propercy  4 


lOZ  per  period  or  week) , the  position  of  the  Isopols  changes  (see  dotted  line 
In  Figure  3a)  revealing  situations  idiere  the  Increasing  property  does  not  hold. 
However,  for  realistic  discount  rates  (say  20%  per  year  or  less)  we  have  found 

little  change  from  the  undiscounted  systems  for  the  optimal  policy  set. 

2 

When  T < H(R2  - Rj^)  oi/(l  - a)  the  optimal  control  structure  takes  on  a 
different  form.  In  this  case,  property  1 no  longer  requires  the  "do  nothing" 
region  shown  In  figure  3a.  Instead,  property  1 requires  that  for  shop  2, 
whenever  the  backlog  in  what  shop  exceeds  a given  level,  the  optimal  policy 
Is  to  place  all  workers  In  that  shop.  The  optimal  control  structure  for  such 
a system  is  shown  in  figure  3b. 

In  figure  4 the  consequences  of  the  Inertia  property  for  a 3 worker  system 
are  noted.  The  four  planes  shown  give  examples  of  isopol  configuration  when 
the  number  of  workers  in  shop  1 at  the  previous  stage  are  3 for  the  top  plane, 

2 for  the  plane  below,  1 for  the  plane  below  that,  and  0 for  the  bottom  plane. 
Note  that  In  those  regions  of  the  state  space  where  the  optimal  policy  Is  to 
Increase  the  number  of  workers  In  shop  1 Isopols  correspond  to  those  In  the 
bottom  plane.  When  the  optimal  policy  Is  to  decrease  the  number  of  workers 
In  shop  1,  Isopols  correspond  to  those  In  the  top  plane. 

To  see  why,  consider  the  following  example:  Suppose  that  the  optimal 
policy  at  point  (B,  3,  0)  Is  (1,  2).  Then  points  (B,  2,  1)  and  (B,  1,  2) 

fulfill  the  first  requirement  of  Theorem  3 (i.e.,  d^^  —*^1  L 

optimal  policy  In  any  of  the  three  points  Is  to  decrease  the  number  of  workers 
In  shop  1,  the  second  condition  of  Theorem  3 (d^  — Pp  satisfied  and  p^ 
must  equal  p^.  This  does  not  yet  guarantee  that  p^  ■ p^.  The  only  way  to 
establish  this  Is  to  show  that  the  second  condition  of  Theorem  3 Is  Indeed 
satisfied. 

Theorem  4 can  be  used  to  do  this.  Suppose  that  d^  < p^  then  p^^  ''  *^1  ^ P^’ 

But  Theorem  4 stipulates  that  Pj^  ^ p|,  a contradiction.  Thus,  d^  ^ pj^  and 
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(1.0) 


tolifin  whan 


Flgur*  4;  Xh«  Effect  of  Inertia  on  a TVo-Shop 

Three  Worker  Parallel 


r 

1 

f 

\ must  equal  p'.  To  summarize,  if  p is  optimal  at  (B,  3,  0),  and  if  3 ^d'  ^ P,  . 

! _ - - 

[ then  p is  optimal  at  (B,  dj^,  dp.  Similar  logic  shows  that  if  p is  optimal  at 

i _ 

[ (B,  0,  3)  and  if  0 ^ d|^  <_  p^^,  then  p is  optimal  at  (B,  dp  dp. 

The  "undetermined"  regions  are  undetermined  in  the  sense  that  the 
Inertia  property  itself  cannot  be  used  to  determine  the  policies  in  the 
regions.  Properties  1 and  2 can  be  used  to  fix  portions  of  the  "undetermined" 
regions.  In  addition,  it  is  easy  to  see  that  if  the  increasing  property 
holds,  the  "undetermined"  regions  are  do  nothing  (i.e.,  maintain  the  present 
worker  allocation)  regions.  Assuming  that  the  increasing  property  holds  (when 
the  shop  system  is  modeled  as  an  undiscounted  stationary  process)  an  efficient 
algorithmic  approach  can  be  used  to  determine  optimal  strategies. 

Consider  a 2-shop  W-worker  system  where  the  objective  is  to  minimize  the 
total  discounted  operating  cost  over  an  infinite  horizon.  Applying  the  well 
known  Howard  [13]  policy  iteration  technique  in  a straight-forward  fashion 
requires  at  each  iteration,  for  each  state,  a search  over  all  possible  policies. 
Assuming  (for  the  moment)  the  existence  of  the  Increasing  property  a policy 
iteration  may  be  completed  in  the  following  manner.  Determine  policies  for 
Che  states 

(B,  0,  W),  (B,  W,  0) , B ^ 0. 

Policies  for  the  states 

(B,  1,  W-1),  (B,  2,  W-2),  ...,  (B,  W-1,  1)  B ^0 
are  determined  directly  by  applying  Che  inertia  and  increasing  properties. 

Use  of  this  property  reduces  computational  effort  from  one  which  is  quadratically 
expanding  in  W to  one  which  is  linearly  expanding.  In  addition,  further 
economies  can  be  effected.  Starting  Che  Iteration  at  states  (0,  0,  W)  and 
(0,  W,  0)  and  moving  to  states  with  successively  larger  backlogs  Che  increasing 
property  can  be  used  to  limit  the  policy  search  for  the  (B,  0,  W)  and  (B,  W,  0) 
states. 


Once  Che  algorithm  converges,  one  additional  iteration  checking  for  poten- 
tial optimal  (decreasing)  policies  insures  the  existence  of  the  increasing 
property.  It  should  be  noted  parenthetically  (recall  the  earlier  discussion 
of  Property  5)  that  Instances  where  the  property  does  not  hold  (for  the  sta- 
tionary undlscounted  system)  are  rare,  and  possibly  non-existent. 

It  is  possible  to  quantify  precisely  the  Inertia  property's  ability  to  re- 
duce the  size  of  a problem.  Suppose  that  a two  shop  system  has  a limited  state 
space  where 

0 ^ ^ 1000  mhrs.  and  0 ^ ^2  — mhrs. 

(Of  course  the  problem  as  formulated  assumes  that  the  backlog  may  take  on  any 
value  no  matter  how  large.  However,  it  is  not  unreasonable  to  place  an  upper 
bound  on  allowable  backlog  thereby  making  a computational  solution  possible.) 
Suppose,  further,  that  for  computational  purposes  the  state  space  has  been 
divided  into  40x40  manhour  blocks.  For  a fixed  previous  stage  worker  allo- 
cation there  will,  therefore,  by  25x25  or  625  states  associated  with  that 
allocation  alone.  If,  for  example,  there  are  30  total  workers  in  the  system, 
there  will  then  by  625x30  or  18,750  total  states.  But  by  utilizing  Che  inertia 
property  the  problem  can  be  reduced  to  one  where  only  625x2  or  1,250  states 
must  undergo  policy  search. 

It  should  be  noted  that,  for  computational  purposes,  it  does  not  appear 
necessary  to  be  overly  concerned  with  the  existence  of  the  increasing  pro- 
perty. Our  experience  has  been  that  the  inertia  property,  and  Property  2 
account  for  as  much  as  95%  or  more  of  the  policies  in  the  interior  states. 

Extensions  of  Results  to  N-Shop  Systems 

Extensions  of  the  results  presented  here  to  N-shop  systems  are  generally 
straightforward.  The  reader  is  referred  to  Haas  [12]  which  contains  a complete 
discussion  of  Che  extension  to  M shops. 
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I To  sunmarlze,  we  have  shown  that  parts  of  the  state  space  for  the  two 

j 


shop  optimal  control  problem  can  be  sudlvlded  Into  regions  where  either; 
the  optimal  control  policy  Is  known  exactly  (as  with  properties  1 and  2) , or 
where  the  process  for  finding  a solution  Is  of  reduced  complexity  (as  with 
property  4).  Furthermore  Property  3,  by  showing  that  all  optimal  policies 
are  completely  determined  when  optimal  policies  are  known  for  only  (B,  W,  0) 
and  (B,  0,  W) , drastically  reduces  computer  storage  and  time  requirements 
when  solution  of  a problem  by  computational  methods  Is  necessary.  This  Infor- 
mation should  prove  to  be  more  and  more  useful  as  production  facilities  move 
to  higher  degrees  of  systematic  or  automated  control. 

Even  If  policies  are  not  determined  precisely,  a general  knowledge  of 
the  layout  of  the  control  structure  can  be  used  to  advantage.  The  two  fund- 
amentally different  forms  (described  by  Property  1)  which  the  optimal  control 
structure  may  take  are  a case  In  point.  By  utilizing  the  Information  In 
Property  1,  the  shc7>  system  controller  can  determine  from  shop  parameters 
which  basic  form  of  control  to  apply  to  a system,  and  then  act  accordingly. 
Su-'.h  Insights  Into  control  structure  can  be  very  valuable  when  exercising 
shop  control. 
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PART  II 


Introduction 

Before  beginning  this  discussion  of  the  formulation  of  series  shop  systems 
the  reader  Is  referred  to  the  introductory  and  formulation  discussion  In  Part  1 
(parallel  shops) . 

The  formulation  for  the  series  shop  system  Is  quite  similar  to  that  for 
the  parallel  system.  In  fact,  the  cost  structure  for  the  series  system  is 
Identical  to  the  parallel  system.  The  only  difference  in  the  two  is  the  flow 
of  work  from  shop  to  shop.  In  the  series  system,  work  which  exists  one  shop 
enters  the  next  shop  in  the  series  as  input  where  it  awaits  processing  in  that 
shop  (see  Figure  1) . It  is  assumed  that  work  completed  in  shop  1 during  the 
period  is  not  available  to  shop  2 until  the  beginning  of  the  next  period. 

There  are  no  backlog  limitations  for  any  shop,  and  input  to  the  first  shop  is 
the  only  stochastic  element  of  the  system. 

The  difference  in  workflow  (parallel  vs.  series)  can  be  completely  ac- 
comodated by  the  transition  function  f^(mlB)  so  that  the  recursive  equation 
for  the  series  system  appears  exactly  the  same  as  for  the  parallel  system. 

That  is: 

$(B,  d)  * min  {t|p^  -d^|+H  ^B.  + ar'  f^(m|B)  ♦(m,  p]<dm}. 

o^i^w  ^ i-1  ^ Jo 
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Note  that  the  only  difference  In  the  parallel  system  formulation  Is  the  Im- 
plicit fact  that,  f^(m|B)  now  reflects  the  workflow  movement  of  a series  system. 
Otherwise,  the  symbolism  used  above,  and  throughout  Part  2,  Is  the  same  as  that 
defined  In  Part  1. 

The  Structure  of  the  Optimal  Control  Policy 

In  this  section  three  fundamental  properties  of  the  optimal  control  policies 
for  an  Infinite  horizon  system  will  be  determined.  These  properties  will  be 
used  to  reveal  the  basic  nature  of  the  optimal  control  structure  for  series 
system. 

Property  1: 

The  first  property  to  be  established  shows  that  If  the  transfer  cost  Is  less 
than  a specified  critical  number,  then  when  backlog  Is  sufficiently  great,  the 
optimal  policy  Is  to  reallocate  manpower  so  that  all  workers  are  in  shop  2.  If 
T Is  greater  than  the  critical  number.  It  Is  never  optimal,  no  matter  what  the 
backlog  level,  to  transfer  workers. 

Before  beginning  the  proof  for  property  1,  two  necessary  lemmas  will  be 
established. 

Lemma  1; 

«(Bj^  + 0,  B^,  d)  - <KBj^,  B2,  d)  (1) 

(The  same  property  holds  In  B2.) 

Proof: 

The  proof  follows  that  for  Lemma  1 In  Part  1.  (As  In  Part  1,  It  Is  easy 
to  show  that  In  the  limit  as  B^  <»,  the  Inequality  of  Lemma  1 becomes  an 
equality.) 
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Lesmna  2: 


Proof : 


|♦(B,  d)  - HB,  d')|  £ |d^  - d|l  T 


The  proof  follows  Chat  for  Lemma  2 in  Parc  1. 

Theorem  1. 

Let  R ■ Max  {R^^,  R2}.  (Note  that  R^  Is  the  period  man  hour  capacity  of  shop  1) 
Define 

n ■ + n^. 

1.  When  n^^  can  be  selected  large  enough  that 

1 “i 

“l‘  . a -^(In^RH) 

r .1  1 


HR,  I ia^  - 


T < 


1-1 


1 - a . 


(2a) 


1 + a 


and  If  n2  Is  selected  so  that 


2Rao 


n2+l 


(1  - a) 
and  so  that. 


2 - 


< T 


(2b) 


t n,*!  O- I a, 

Ta.»)<ilR,  I 2™ 


1-n, 


1 - a 


(2c) 


Then,  when  B2  ^ n * R2  * W,  it  Is  optimal  to  place  all  workers  In  shop  2 for 
the  first  n^  stages.  (In  other  words.  If  the  backlog  In  shop  2 Is  big  enough,  and 

T < HR2a/(l  - a)^  , 

then  It  Is  optimal  to  place  all  workers  In  shop  2.) 
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2.  When 


k 


L 


T > HR2a/(l  - a)^  (2d) 

It  is  always  optimal,  no  matter  what  the  backlog  level,  to  hold  the  worker 
allocation  of  the  previous  stage  constant. 

Proof ; 

It  will  first  be  assumed  that  it  is  never  optimal  in  the  first  n^  stages 
to  decrease  the  number  of  workers  in  shop  2.  Further,  it  will  be  assumed  Chat 
if  it  is  optimal  (in  the  first  n^  stages)  to  Increase  workers  in  shop  2,  that 
Increase  must  take  place  at  stage  1.  These  assumptions  leave  only  two  candi- 
date optimal  policies,  either  maintain  the  worker  allocation  of  the  previous 
stage,  or  Increase  (at  stage  1)  the  number  of  workers  in  shop  2.  The  first  two 
parts  of  the  proof  will  Involve  a comparison  of  these  candidate  policies. 

In  part  1 it  will  be  shown  chat  (given  Che  above  assumptions)  and  given 
that  (2a)  holds  it  is  optimal  to  Increase  Che  ntimber  of  workers  in  shop  N.  In 
fact,  it  will  be  shown  that  it  is  optimal  to  place  all  workers  in  shop  2.  In 
part  2 it  will  be  shown  Chat  given  Chat  (2d)  holds  it  is  always  optimal  to 
maintain  Che  worker  allocation  of  the  previous  stage.  In  part  3 the  proof  will 
be  completed  by  showing  that  each  of  Che  Initial  assumptions  must  hold. 

Part  1;  Assume  that  inequality  (2a)  holds.  Now  fix  an  input  realization  and 
define  option  1 as  follows:  For  the  first  n^  stages,  policy  d'  (d^  ^ 0)  is  in 
effect.  Thereafter,  the  system  is  optimally  controlled.  The  expected  cost  of 
Che  system  is 


*'(B,  d)  - C*^'  + C^'  + a ^ C’  , 

nj^  nj^ 


y 


t' 

where  is  the  expected  discounted  transition  cost  through  n^ 


C Is  the  expected  discounted  holding  cost  through  n.  transitions,  and 
“l 

Is  the  expected  discounted  cost  Incurred  for  all  stages  after  n^. 

For  the  same  Input  realization  define  option  2 as  follows:  The  worker 
allocation  for  the  first  n^  stages  Is  d".  d"  Is  the  same  as  d'  except  that 
d!^  >■  d^  **  1 and  the  worker  lost  to  shop  2 Is  added  to  shop  1. 

This  policy  remains  In  effect  for  the  first  n^  stages.  Thereafter,  the 
system  Is  optimally  controlled.  The  expected  cost  for  option  2 Is 


- - t"  h"  1 

f"(B,  d)  - C + C + o C" 


ni  n^ 


nj^+1  ' 


For  any  Input  realization,  the  backlog  levels  for  option  1 at  stage  n^  + 1 
and  for  option  2 at  stage  n^  + 1 are  limited  In  separation.  The  limitation  Is 
as  follows:  If, 

■ backlog  In  shop  1 for  option  1 at  stage  n^^+l, 

B'^  ■ backlog  In  shop  1 for  option  2 at  stage  n^^  + 1, 


then. 


|B^  - B'^l  < 2n^R  . 


From  Lemma  1 It  can  easily  be  shown  that. 


Then, 


IiKB",  d')  - ®(B",  d")  1 1 |d^  - d^|T  - T . 


®(B',  d')  - $(B",  d")|  1 |*(B',  d’)  - $(B",  d') 


+ l-td",  d')  - *(B",  d") 


2n  RH 
— 1-a 
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h'  r 1 

C + HR-  ) ia  . 

“l  ^ i-1 


i 


(5) 


Adding  (4)  and  (5) , 


Mtiltlplylng  by  the  denominator  gives,' 


L 


Ui-l 


^ a ^(2n  RH) 

T(1  + a ^)  < HR,  I ia r ± 

i»l 


Rearranging  yields. 


ni-1 


n 2n  RH  . 

a (-tV  + T)  < hr,  I ia^  - T. 
1-a  - 2 ^ 


(9) 


(8)  and  (9)  imply  that. 


,h" 


l„h" 


In. 


C"  + C + a C > C + C + a ‘C’  , . 

n^^  n^  n^^  — n^^  n^^  n^^+l 


(10) 


Eq.  (10)  shows  that  ^ the  policy  selected  at  stage  1 is  maintained  foe  the 
first  n^  stages,  and  if  (2a)  holds  policy  d'  provides  a lower  expected 
cost  than  policy  d".  Repeated  application  of  this  result  shows  that 
the  policy  selected  at  stage  1 is  ibalntalned  for  the  first  n^  stages,  and 
if  (2a)  holds,  the  expected  return  when  all  W workers  are  placed  in  shop  2 
at  stage  1 is  less  than  or  equal  to  the  expected  return  with  any  other 
policy  selected. 


Part  2;  Note  that  HR,  I io  represents  an  upper  bound  on  holding  cost 


i-1 


reduction  which  can  result  from  the  transfer  of  a single  worker.  Clearly, 
then,  if 


T > HR,  I io^  - HR,a/(l-a)^, 


i-1 


it  is  never  optimal  to  make  a worker  transfer. 

Part  3;  The  validity  of  the  initial  assumptions  will  now  be  established. 

This  is  done  by  comparing  the  assumption  with  the  alternative  policy, 
using  the  relationships  in  the  theorem  statement. 


44 


nu]id>er  of  workers  In  shop  2 is  nonoptlmal. 

Fix  an  Input  realization.  Suppose  that  stage  X+1,  2 ^ X+1  £ n^^ 
is  the  first  point  where  the  optimal  policy  is  to  Increase  the  number  of 
workers  in  shop  2,  and  suppose  further  that  the  shops  are  optimally  control- 
led thereafter.  Define  this  as  option  1.  Define  option  2 as  follows: 

Option  2 is  the  same  as  option  1 except  that  the  Increase  in  shop  2 Is 
made  one  stage  earlier,  at  stage  X. 

Since  shop  2 cannot  run  out  of  work. 


C“  - HR  I 
“l^2  ^ 1-1 


The  difference  In  transfer  costs  for  the  two  options  can  be  expressed  by 
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(1^)  and  (15)  imply  that 


h"  t"  “1^2  " h'  t' 

C + C + a C , < C + C 

nj^+n2  “i'*^2  n^^+n^+l  — nj^+“2 

n +n 

+ a ^ ^ C'  ,, 

ni+n2+l 

Assumption:  Any  policy  change  at  stages  1 through  n^  which  decreases 

the  number  of  workers  In  shop  2 Is  nonoptlmal. 


Fix  an  Input  realization. 

Define  option  1 as  follows:  At  stage  X policy  d'  Is  In  effect. 

Policy  d*  decreases  the  number  of  workers  which  were  In  shop  2 at  the 
previous  stage  and  Increases  the  number  of  workers  In  shop  1*  Option  2 
Is  defined  as  follows:  At  stage  X the  policy  In  force  Is  d"  ■ - 1»  d^  + 1) 

Since  shop  2 cannot  nm  out  of  work. 


h”  h' 

C ^ ■ C . 

nj^Tn2  °l"™2 


Ui+nj-l 


- hr,  I 


Difference  In  transfer  costs  for  the  two  options 


1C"  - Ta"  -"(l-a)  < 

By  arguments  similar  to  those  used  earlier.  It  Is  easily  seen  that 

“1^2  „ “l''^2  , g ^ ^ 2RH  , 

° nj^+n2+l  “ ^ nj^+n2+l  1 - o 

Adding  (16),  (17),  and  (18)  to,;ether  and  using  (2c), the  same  logic  used  In 
the  previous  assumption  shows  that 

h”  t"  “l'^2  h'  t' 

+ C . + o C'  . , < C . + C . 

“l'^2  “l'^2  nj^4ii2+l  ~ “i‘^2  “l'^2 

n +n 

+ 0 C".  . 1 


Q.E.D 
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} 

Property  2; 

Just  as  backlog  can  be  so  large  that  the  optimal  policy  Is  specified, 
backlog  levels  also  may  be  so  low  that  the  optimal  policy  1 s a 1 s o 
specified. 

Before  proving  this  property,  one  lemma  must  be  established. 

Lemma  3; 

$(B,  d)  <_  $(Bj^  + B2  - + 02,  d),  0 0^  B^,  1 - 1,  2. 

Proof:  Define 

V * (B,  d) 

p'  - (Bj^  + 0j^,  B2  - 0j^  + 02.  d) 

It  Is  easy  to  show  that  for  any  given  Input  realization.  If  the  optimal 
policies  generated  for  starting  at  point  p'  are  used,  the  total  backlog 
(^B^)  at  any  stage  of  the  sequence  starting  at  point  p will  always  be  the 

same  or  less  than  the  total  backlog  level  generated  from  starting  at  point  p'.  j 

Thus,  the  total  expected  cost  of  operating  the  system  for  any  Input  1 

j 

realization  must  be  more  If  the  system  Is  presently  at  p'.  j 

Q.E.D.  I 

With  this  lemma  established.  It  Is  possible  to  proceed  with  the 
following  theorem: 

Theorem  2: 

If  the  state  of  the  system  (B,  d)  Is  such  that  B^^  £ *^1^1  ^ 
optimal  policy  Is  to  maintain  the  present  allocation  d. 

Proof : 

Consider  what  happens  to  the  backlog  levels  at  a point  In  time  Immediately 
before  the  upcoming  Input  surge.  Assume  that  with  policy  d the  backlog  level 
would  have  been  B Immediately  before  the  Input  surge. 
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If  a worker  from  shop  1 is  removed,  the  net  effect  is  to  Increase  the 
backlog  in  a shop  Iby  and  to  decrease  the  work  In  shop  2 

by  If  a worker  Is  removed  from  shop  2 the  backlog  in  shop  2 Is  Increased 

by  82(0  ^ 02  — ^2^  shop  I's  backlog  is  not  affected.  The  backlog  in  the 
shop  which  received  the  removed  worker  will  remain  unchanged  since,  by  the 
theorem  assumption,  each  shop  has  enough  workers  to  complete  all  work  which 
was  in  the  shop  at  the  previous  transition. 

Thus,  for  any  policy  p which  is  different  than  d,  the  backlog  immediately 
before  the  upcoming  input  surge  will  take  the  form: 

+ dy  ®2  ■ ®1  ®2^' 

Thus,  from  Lemma  3,  it  is  easy  to  see  that 


$ (m,  d)  f^(mlB)  dm  ^ ($  (m,  d)  f^(m|B)  dm. 


Since  policy  p also  entails  transfer  costs,  it  follows  that  it  is  cheaper  to 
maintain  the  present  policy  d. 


Q. 


I 


E.D. 


Property  3i 

Property  3 is  established  for  the  series  system  in  exactly  the  same 

I 

I 

way  as  for  the  parallel  system.  Theorems  3 and  4 of  Part  1,  which  established 
property  3 for  parallel  shops,  are  restated  here. 

Theorem  3; 

Consider  two  distinct  points  in  the  shop  state  space  (B,  d)  and  (B,  d'). 

% 


i 


8 


ii 


Let  p be  optimal  at  (B,  d)  and  let  p'  be  optimal  at  (B,  d').  Furthermore, 
for  all  1,  let  either 

d^  £ d^  ^ and  d^  £ p^ 

or  i ^ dj^  ^ p^ 

Then  p*  Is  also  optimal  at  (B,  d) , and  p Is  also  optimal  at  (B,  d'). 

Proof: 

The  proof  Is  Identical  to  that  for  theorem  3 In  Part  1. 

Theorem  4; 

Fix  some  point  B In  backlog.  Let  p be  the  optimal  policy  when  the 
worker  allocation  for  the  previous  transition  Is  S.  Let  p'  be  the  optimal 
policy  at  B when  the  worker  allocation  for  the  previous  transition  Is  S + Q, 
and  where  Q Is  a vector  fdiose  elements  may  be  positive  or  negative. 
Furthermore , let 

< p^  V 1 such  that  < p^ 

> Pj^  V 1 such  that  p^  < 

Then,  for  all  1 

^ p^  when  < p^,  and 

^ when  > p^. 

Proof: 

The  proof  is  Identical  to  that  for  theorem  4 in  Part  1. 


! 
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Geometric  Interpretation: 

The  properties  developed  yield  a great  deal  of  Information  about  the 
optimal  control  structure  of  a series  shop  system.  Figure  2 depicts  the 
general  structure  of  the  optimal  control  policies.  Property  1 Indicates 
what  optimal  policy  will  be  In  force  In  shop  2 when  backlog  In  that  shop 
becomes  large.  When 


T < HR2a/(l-a)^, 

as  It  Is  for  the  system  represented  In  Figure  2.  That  policy  will  be  to  place 
all  workers  In  shop  2.  If  T Is  larger,  the  problem  Is  trivial.  (No  worker 

Is  ever  transferred.) 

f ■ 

Property  2 Indicates  that  In  an  area  close  to  the  origin,  the  optimal 
policy  Is  to  maintain  the  policy  of  the  previous  stage. 

Property  4 shows  that  In  a region  where  backlog  In  shop  1 Is  large  the 
optimal  control  structure  can  be  found  by  solving  a 1 shop  problem. 

The  effect  of  property  3 on  the  series  shop  optimal  control  structure 
Is  shown  In  Figure  3.  The  four  planes  shown  give  examples  of  Isopol 

configuration  when  the  number  of  workers  In  shop  1 at  the  previous  stage 
are  3 for  the  top  plane,  2 for  the  plane  below,  1 for  the  plane  below  that, 
and  0 for  the  bottom  plane.  Note  that  In  those  regions  of  the  state  space, 
where  the  optimal  policy  Is  to  Increase  the  number  of  workers  In  shop  1, 
Isopols  correspond  to  those  In  the  bottom  plane.  When  the  optimal  policy 
Is  to  decrease  the  number  of  workers  In  shop  1,  the  Isopols  correspond  to 
those  In  the  top  plane. 

Note  that  the  above  properties  can  be  used  In  the  same  manner  as 
those  for  parallel  systems  to  build  efficient  computational  solution 
methods.  (See  the  discussion  In  Part  1.) 
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Figure  General  structure  of  control  policies 


for  a 2-shops  in  series  system  (4  workers) 
where  the  previous  policy  is  (1,  3). 


/ 


1 
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Figure  3:  The  Effect  of  Inertia  on  a 


TVo-Shop,  Three  Worker,  Series 


System 


/ 
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